Not a blocker but I would be careful relying on the hidden property. Maybe this is something we need to expose in the public api?

It's not really "hidden" in the sense that the user shouldn't be able to access it if they're determined. It's just a property that's only really used internally. We're using this to re-derive the variables that were present in the initial variation so that the LLM has a stable reference when trying to replace them

Will this handle things like backof/throttle/retry in the event of 429? Maybe aborting an in-progress optimization is best in cases of permanent errors (eg 401, indicates an invalid token), or even sporadic errors like 403, where certain requests fail, and others succeed, but it would lead to incorrect results?

The api key gets checked up-front with the fetch for model configs etc. so shouldn't error unless it's revoked midway through. Given that, I'd say it makes sense to fail on any 401s here.

I'll add some retry logic for other status codes up to a max and then fail the optimization if we hit max retries

What does this do? Does it allow the LLM to emit arbitrary keys, which then get persisted on agent_optimization_result.parameters and, on auto_commit, pushed as the new AI Config variation's parameters? It looks like we only expect certain values here?

Maybe after extract_json_from_response, filter current_parameters to a known allow-list of LLM call parameters — e.g., {temperature, top_p, max_tokens, presence_penalty, frequency_penalty, stop} — and drop unknown keys with a warning log. This also belongs in the server-side validation on POST /ai-configs/{config_key}/variations.

Let me look into this, it may only exist for posterity at this point (moved away from tool-based validation on these since some providers didn't play nicely with it).

In the case we need to keep it, we unfortunately cannot know what is valid in this parameters object given that the user provides their LLM call. Each provider's parameters differ so we could only allow-list if we knew up-front which provider they were using

This is not really safe to log, as it likely contains sensitive info

Do we need this? Why such an old version? Is having cool names for variation keys worth the risk of this dependency? Would the 'cool' names even be useful/meaningful, vs a default date-based name?

Discussed in review meeting; will just hand roll this as its pretty simple to implement

the flexibility here is neat, but it increases the risks of adversarial JSON smuggling.

Would it be possible to get metrics from how many times the fallback methods are used?

Maybe if a fallback method was used, we should check to see if there are unexpected keys in the resulting object, and WARN-log it, to hopefully alert the customer that something is amiss?

Yeah this is basically just fallback parsing if the LLM decides to respond with something like:

Here is your JSON:
{...}

Unfortunately since we don't know which provider the user is using for this we can't enforce any structured output (for those that even support it).

I think it's worth emitting a warn on

I won't block on this but all other launchdarkly python packages have the launchdarkly-* name.